Open Science in Water Resources with the Jupyter Notebooks.
With the ever increasing availability of data and ever more complex computational models, the computational and coding needs of water resources professionals is greater now than ever. However, water resources community currently has no ‘lingua franca’for scientific computing and coding is being done in a wide variety of specialized environments computer languages. This has created a situation where the complexity of these computing environments is inhibiting many from being able to easily share their code with colleagues across various countries and organizations. The open source community is addressing this issue by developing innovative programming tools and environments which allow for a universal interface for computing in a variety of languages. Jupyter Notebooks, developed by the Jupyter Project, enable the combination of rich multimedia documentation and interactive code within a standard web browser interface, they facilitate the setup and sharing of customizable computing environments in a variety of programing languages, and eliminate the need for expensive licensing in many scientific computing applications. This presentation introduces Jupyter Notebooks as applied to a variety of problems in water resources engineering. Examples notebooks include applications for coastal and inland water resources projects as well as statistical and data analysis tools.
What do we mean by Open Science?¶
...¶
Open Data and Open Tools!¶
Open Data Sources
NOAA¶
USGS¶
By Open Tools we really mean Free Open-Source Software
"Free and open-source software (FOSS) is computer software that can be classified as both free software and open-source software. That is, anyone is freely licensed to use, copy, study, and change the software in any way, and the source code is openly shared so that people are encouraged to voluntarily improve the design of the software." - Wikipedia
Why is FOSS important?¶
Software that is both "free as in beer" and "free as in speach" gives users of computing hardware (read 'almost everyone!') freedom of choice in the software they use and how they use their system. This freedom has allowed for the rapid development of technologies, such as the internet and more recently the internet of things, that allow us to interact with computers in all of our lives on a day to day basis.
The Python design ethos is focused on simplicity, consistency, and readibility. This is summarized in the "Zen of Python"
import this
Because of the focus on simplicity, consistency, and readability Python is increadibly easy to learn for both begining programmers and old hands alike.
(another Python easter egg)
$ import antigravity
*'print "hello world!"' is Python 2 parlance, in Python 3 this is 'print("hello world!")'.
print("hello world!")
Where is Python used?¶
Simple answer - nearly everywhere.¶
Python is a popular programming language used primarily for web development, education, and scientific computing. However, due to its popularity, flexibility, and gentle learning curve there are Python modules for almost anything. (And more new ones everyday)
Think of Python as glue. It gets everywhere and holds things together.
Everyone here should know that it's also built into ArcGIS. But did you know it's part of QGIS as well?
Python is popular!¶
So, how can water professionals use Python for science at work?¶
The SciPy Stack!¶
The SciPy Stack consists of 6* modules that extend the capabilities of Python into the realms of Matlab, R, and SAS.
These include the following:
- SciPy -------> scientific computing in python including signal processing and optimization
- Numpy -------> basic array manipulation
- Pandas ------> data manipulation, R-style dataframes
- SymPy -------> Symbolic math ala Mathematica/Maple
- Matplotlib ---> visualization and plotting
- scikit-learn--> machine learning (not strictly part of the SciPy stack)
- IPython -----> write and run python code interactively in a shell or a notebook
-- Note: the notebook functionality of IPython was recently spun off to Jupyter.
What is Jupyter?¶
... This is Jupyter!¶
Project Jupyter was born out of the IPython Project in 2014 as it evolved to support interactive data science and scientific computing across all programming languages.
The name Jupyter is a portmanteau of the julia, Python, and R programming languages.
However, rapid development in the past two years means that the notebook now supports more that 60 language kernels! Including: Matlab, R, Ruby, JavaScript, Bash, and C++.
The Jupyter Notebook is a multi-language web-based IDE that integrates the multimedia capabilities of the modern web, the elegant typography and equation editing capabilities of LaTeX, and the powerful computing capabilities of numerous popular programming languages within a single interface.
Built in documentation
%quickref
Shell commands
%ls
%mkdir example
%cd example
%ls
%cd ..
%rmdir example
#%ls
%%bash
for i in `seq 1 10`;
do
echo $i
done
Quickly and easily profile your code
import numpy as np
%timeit np.linalg.eigvals(np.random.rand(100,100))
%%timeit a = np.random.rand(100, 100)
np.linalg.eigvals(a)
2. LaTeX¶
You can pass raw LaTeX test as a string to the Math object:
from IPython.display import Math
Math(r'F(k) = \int_{-\infty}^{\infty} f(x) e^{2\pi i k} dx')
Or call LaTeX with a cell magic
%%latex
$G=\frac{m_{1}*m_{2}}{d^{2}}$
3. HTML wizardry¶
Audio¶
from IPython.display import Audio
Audio(filename="sweeter_than_wine.m4a")
Audio(filename="Kalimba.mp3")